The Linkscape web index updated this past Wednesday, December 8. You can see new link data for sites and pages via Open Site Explorer, the mozBar, the Web App and Linkscape Classic. This is also one of our freshest and fastest index updates (from the prior update on November 16, it’s only been 22 days), so the links you’ll see were crawled at the end of November and processing occurred in the beginning of December.
This index is slightly smaller than others, but oddly, it appears to be due to some extra decay on the web; we’ve requested as many pages as normal but just haven’t seen as many still alive. Perhaps the holidays are the time to put old sites on ice, as last year appeared to show a somewhat similar pattern (tough to confirm whether this is Linkscape’s crawling behavior vs. a true representation of the web, though).
Index 34’s stats are:
- 39,821,634,471 (39.8 Billion) Pages
- 420,451,251 (420 Million) Subdomains
- 104,307,322 (104 Million) Root Domains
- 387,736,255,184 (388 Billion) Links
- 2.08% of All Links are Nofollowed (down 0.02% from November)
- 57.66% are internal (up from 56.99% in November)
- 42.34% are external (down from 43.01% in November)
- 5.91% of pages have rel=canonical (up from 5.88% in November)
- 62.11 links/page on average (down from 62.28 in October)
Since it’s the end of the year, I thought it would be interesting to take a look back at some of the Linkscape data from the beginning of 2010 up through this latest update. Just remember that this comes from Linkscape’s indices only, so the entire web is represented, though a substantive portion is included (many tens of billions of pages per index).
Rel=Canonical Use in 2010
Links / Page in 2010
Nofollow Distribution in 2010
Quantity of Subdomains per Root Domain in 2010
Average Percent of Links to a Site that Came from the Same C-Block of IP Addresses in 2010
While we’re excited with how far Linkscape has come in 2010, there’s a lot more progress to be made and a lot of effort going into it. In 2011, expect to see even faster updates, recursive crawling (meaning new pages on the web go into indices at an increased clip), much larger indices (25-50%+ larger) to reach deep down into those corners of the web we miss today, and more views on the data.
In the meantime, we invite you to use the tools above and the Awesome FREE SEOmoz API, and tell us what else you’d like to see in the future.
Finally, SEOmoz PRO is getting nominations for two categories of TechCrunch’s Crunchies awards. It would mean a lot to us if you’d drop by and support our nod for Best Internet Application and Best Technology Achievement. TC takes one vote/day until midnight on December 24, so if you feel inclined, feel free to vote daily. Thanks!
